Analyzing Multimodal Communication around a Shared Tabletop Display
نویسندگان
چکیده
Communication between people is inherently multimodal. People employ speech, facial expressions, eye gaze, and gesture, among other facilities, to support communication and cooperative activity. Complexity of communication increases when a person is without a modality such as hearing, often resulting in dependence on another person or an assistive device to facilitate communication. This paper examines communication about medical topics through Shared Speech Interface, a multimodal tabletop display designed to assist communication between a hearing and deaf individual by converting speech-to-text and representing dialogue history on a shared interactive display surface. We compare communication mediated by a multimodal tabletop display and by a human sign language interpreter. Results indicate that the multimodal tabletop display (1) allows the deaf patient to watch the doctor when she is speaking, (2) encourages the doctor to exploit multimodal communication such as co-occurring gesture-speech, and (3) provides shared access to persistent, collaboratively produced representations of conversation. We also describe extensions of this communication technology, discuss how multimodal analysis techniques are useful in understanding the affects of multiuser multimodal tabletop systems, and briefly allude to the potential of applying computer vision techniques to assist analysis.
منابع مشابه
A Framework for Bimanual Inter-Device Interactions
A shared interactive display (e.g., a tabletop) provides a large space for collaborative interactions. However, a public display lacks a private space for accessing sensitive information. On the other hand, a mobile device offers a private display and a variety of modalities for personal applications, but it is limited by a small screen. We have developed a framework that supports fluid and sea...
متن کاملSetting the stage for embodied activity: scientific discussion around a multitouch tabletop display
Multitouch tabletop displays are a promising technology for supporting small group discussion and learning activities. Based on a view of human activity as multimodal, distributed, and embodied, we examine the pedagogical affordances of interactive tabletop technology, how it supports small group scientific discussion, and compare it with use of traditional paper-based study materials. We analy...
متن کاملSystem Guidelines for Co-located, Collaborative Work on a Tabletop Display
Collaborative interactions with many existing digital tabletop systems lack the fluidity of collaborating around a table using traditional media. This paper presents a critical analysis of the current state-of-the-art in digital tabletop systems research, targeted at discovering how user requirements for collaboration are currently being met and uncovering areas requiring further development. B...
متن کاملUnraveling Students' Interaction Around a Tangible Interface Using Gesture Recognition
In this paper, we describe techniques to use multimodal learning analytics to analyze data collected around an interactive tangible learning environment. In a previous study [4], we designed and evaluated a Tangible User Interface (TUI) where dyads of students were asked to learn about the human hearing system by reconstructing it. In the current study, we present the analysis of the data colle...
متن کاملTea Table Mediator: A Multimodal Ambient Display on the Table Exploiting Five Senses Convergence
In the past, many tabletop systems have used the traditional table augmented with digital technologies to enhance collaborative works such as faceto-face or remote meeting and working. In this paper, we describe the Tea Table Mediator, a multimodal ambient tabletop display designed for five senses convergence experience combined with tea drinking. This ambient display enhances a traditional tea...
متن کامل